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Abstract 

The study utilized a multi-method approach to explore the connection between critical 
thinking and epistemological beliefs in a specific problem-solving situation. Data drawn 
from a sample often third-year bioscience students were collected using a combination of 
a cognitive lab and a performance task from the Collegiate Learning Assessment (CLA). 
The cognitive-lab data were analysed using thematic analysis. The findings showed that 
students’ epistemological beliefs were interwoven into their critical thinking: students 
used critical thinking as a tool (1) for enhancing understanding and (2) for determining 
truth or falsehood. Based on this classification, students could be placed in one of two 
qualitative profiles, either (1) thorough processing or (2) superficial processing. The 
results indicated that students who showed superficial processing palmed off justification 
for knowing on authoritative figures. In contrast to previous studies these students did not 
consider knowledge to be absolutely certain or unquestionable. The findings also show 
that students with thorough processing believed knowledge to be tentative and fallible, but 
did not share the relativist view of knowledge where any claim counts because all 
knowledge is relative. All ten students shared afallibilist view of knowledge. 
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1. Introduction 

Critical thinking has been singled out as one of the most important skills for citizens of the twenty- 
first century (Halpern, 2014). Mastering critical thinking is thus a goal that can be found in almost every 
higher education curriculum today. However, recent studies have raised concerns that even though most 
students make significant progress in learning concepts and procedures during their university studies, some 
students show little if any growth in critical thinking (Arum & Roksa, 2011a, 201 lb; Bok, 2006; Pascarella, 
Blaich, Martin & Hanson, 2011). 

In the field of higher education, research on critical thinking has generally focused on the 
development of critical thinking skills (e.g. Arum & Roksa, 2011a; Heijltjes, van Gog, Leppink & Paas, 
2014). Researchers have also highlighted the importance of understanding critical thinking as a social 
activity (e.g. Arum & Roksa, 201 lb; Kuhn, 2005; Moore, 2004; 2013). In this exploratory study we provide 
a multidimensional framework for analysing critical thinking by combining theoretical aspects from 
philosophical, educational and psychological approaches. In our view the concept of critical thinking is 
closely connected to the concepts of ‘knowledge’ and ‘knowing’. Furthermore, we assume that critical 
thinking cannot be formulated by referring to skills alone, but also always involves a disposition to use these 
skills adequately (see Bailin & Siegel, 2003; Holma, 2014; Siegel, 1988). 

Previous research on critical thinking and personal epistemology has frequently applied quantitative 
multiple-choice tests, questionnaires or qualitative interviews (see e.g. Australian Council of Education 
Research, 2001; Heijltjes, van Gog, Leppink & Paas, 2014; Greene & Yu 2014; Lahtinen & Pehkonen, 2013; 
Tremblay, Lalancette & Roseveare, 2012). Recently, many researchers have questioned the reliability and 
adequacy of self-report questionnaires (Greene & Yu, 2014; Elby & Hammer, 2001). As a result, researchers 
have stated that there is a need for studies that assess the performance of students directly (e.g. Elby & 
Hammer, 2001; Hofer, 2004; Stes, Min-Leliveld, Gijbels and van Petegem 2009). At the same time 
researchers have also assumed that one assessment method is not enough to evaluate complex cognitive 
processes such as reasoning (e.g. Baartman, Bastiaens, Kirschner & Vleuten, 2007; Dierick & Dochy, 2001; 
Maclellan, 2004). This study responds to current concerns by exploring students’ critical thinking as well as 
their epistemological beliefs, as elaborated upon below, in a problem-solving situation to which we applied a 
multi-method qualitative approach. A think-aloud method was used as the students worked through an open- 
ended performance task. Our aim is to identify and understand qualitative differences in the critical thinking 
of students and in their beliefs about knowledge, as well as in their personal relationships. 


2. Critical thinking in university-level studies 

Critical thinking is often ‘regarded as fundamental aim of education’ (Bailin & Siegel, 2003, p. 188; 
cf. Dewey, 1910). In a university context critical thinking has an essential role and is an important 
component of the learning outcomes (Bok, 2006). Critical thinking is defined as a process that enables an 
individual to make an informed decision about conflicting claims (Ennis, 1991; Fisher, 2011; Bailin & 
Siegel, 2003). It is purposeful, reasoned and reflective thinking (Ennis, 1991; American Philosophical 
Association, 1990). A critical thinker knows how to assess the strength of evidence and the reasons that are 
relevant to the particular context or type of task, and also shows the disposition to draw on these skills 
(Bailin & Siegel, 2003; Scheffler, 1965, Halpern, 2014). 

Critical thinking is seen as a skilful activity in which a person may be more or less proficient (Fisher, 
2011; Scheffler, 1965). Definitions of critical thinking typically include a list of the thinking skills that 
characterise an ideal critical thinker. For example, Fisher (2011) lists the following: the ability to identify the 
elements in a reasoned case, especially reasons and conclusions; the abilities to identify and evaluate 
assumptions; the abilities to clarify and interpret expressions and ideas; to be able to judge the acceptability, 
especially the credibility, of claims; to evaluate arguments, analyse, evaluate and produce explanations; to be 
able to analyse, evaluate, and make decisions; to draw inferences and produce arguments (see also Halpern, 
2014). University studies require all of these abilities. 
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However, many philosophers have argued that critical thinking cannot be conceptualised merely by 
referring to a prescribed set of skills (Bailin & Siegel, 2003; Holma, 2014; Fisher, 2011; Siegel, 1988, 
Scheffler, 1965; see also Halpern, 2014). It may be that a person has acquired the skills, but does not use 
them (Fisher, 2011). As Holma (2014) has pointed out, it is not enough for students to have critical thinking 
skills; they also need to use these skills effectively. Thus, critical thinking always involves both the essential 
skills or abilities and the disposition to use them (Bailin & Siegel, 2003, Holma, 2014; Siegel, 1988). 

Previous studies have called attention to the fact that students’ critical thinking skills do not always 
develop during university studies (Arum & Roksa, 2011a; Bok, 2006; Pascarella, Blaich, Martin & Hanson, 
2011). Arum and Roksa (2011b) demonstrated in their longitudinal study that a large number of university 
students showed no significant improvement in a range of critical thinking skills, such as reasoning and 
problem solving. However, a recent study by Heijltjes and colleagues (2014) has shown that the combination 
of explicit instruction and practice has proven successful in improving students’ performance in reasoning 
skills. 


3. Knowledge and knowing in critical thinking 

Critical thinking demands a comprehensive use of different types of knowledge (Bok, 2006; Ennis, 
1991). There is a reciprocal relationship between ‘critical thinking’, ‘knowledge’ and ‘knowing’; on the one 
hand, students need knowledge about a phenomenon before they can think about it critically (Halpern, 
2014); on the other hand, students must have the necessary skills to evaluate that knowledge. The concepts 
of‘knowledge’ and ‘knowing’ are thus substantial aspects of conceptualising critical thinking. 

There are several different definitions and classifications of the concept of knowledge. For example, 
philosophical epistemologists usually differentiate amongst three types of knowledge: propositional 
knowledge, procedural knowledge and knowledge by acquaintance (Everitt & Fisher, 1995; Ichikawa & 
Steup, 2012), although there is no consensus on the interpretation of knowledge or on the number of types of 
knowledge (Fenstermacher, 1994). For our purposes the distinction between propositional and procedural 
knowledge has theoretical importance. 

Propositional knowledge is defined as knowing that ‘such-and-such is the case’. This is sometimes 
referred to as factual or declarative knowledge. Propositional knowledge (i.e. ‘knowing that’) is usually 
distinguished from procedural knowledge (i.e. ‘knowing how’) (Ryle, 1949). In philosophical discussions 
propositional knowledge is related to such epistemological concepts as truth, justification, reason and 
evidence (Ryle, 1949; Scheffler, 1965, see also Niiniluoto, 1999; Shope, 2004). Scheffler (1965) argued that 
the ‘knowing that’ attributes of a person may reveal his epistemological orientations, such as the criteria for 
justifying knowing. Empirical research on personal epistemology focuses particularly on these personal 
orientations. 

Procedural knowledge, meaning ‘knowing how’ to do something (knowing how to analyse, knowing 
how to swim, etc.; see Everitt & Fisher, 1995; Shope, 2004), is related to possessing a skill (Scheffler, 1965). 
In this sense critical thinking represents procedural knowledge, which is consistent with the other aspect of 
critical thinking mentioned above. However, several researchers have assumed that procedural knowledge 
always involves some propositional knowledge (i.e. Everitt & Fisher, 1995; Smith 2002; Markowitsch & 
Messerer, 2007). For example, if a person knows how to play chess, he will probably know certain facts (e.g. 
rules) about playing chess. Smith (2002) has emphasized that an individual has a certain skill only when his 
performance reflects both procedural and propositional knowledge. 

In sum, critical thinking involves a disposition to think critically, having the necessary propositional 
knowledge about a phenomenon and having the thinking skills (i.e. procedural knowledge) to evaluate that 
knowledge (cf. Halpern, 2014). 
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4. Students’ epistemological beliefs as premises of critical thinking 

The term ‘personal epistemology’ or, alternatively, ‘epistemological belief is defined as an 
individual’s views of the nature of knowledge and knowing. The term also includes a view of one’s personal 
beliefs as a knower (Pintrich, 2002; Hofer, 2004). The concept of ‘personal epistemology can be described 
along a continuum from less sophisticated to more sophisticated’ ways of knowing (Kaartinen-Koutaniemi & 
Lindblom-Ylanne, 2012, p. 2) or a progress ‘from a state of simple, absolute certainty into a multifaceted, 
evaluative system’ (West, 2004, p. 61). During this process the individual changes from a passive recipient 
of knowledge to an active participant in constructing and evaluating knowledge (Hofer & Pintrich, 2002; 
Kuhn, 2005; King & Kitchener, 2004). Over time epistemological beliefs develop more and more toward 
relativistic beliefs (Hofer & Pintrich, 1997, 2002). 

Previous research on personal epistemology has found that the ability to think critically is embedded 
in a progression of epistemological beliefs (i.e. King & Kitchener, 2004; Kuhn & Weinstock, 2002; Kuhn, 
1999; 2005). Several researchers have hypothesised that students with weak critical thinking skills have an 
absolute view of knowledge. When students move on to the most developed epistemological level, their 
critical thinking tends to improve as well (Bok, 2006; Kuhn, 1999; Kuhn & Weinstock, 2002). It has also 
been demonstrated that students’ epistemological beliefs play an important role in their ability to evaluate the 
credibility of competing claims (Barzilai & Zohar, 2012). 

Whether instruction has any influence on the development of epistemological beliefs is currently 
under discussion (e.g. Valanides & Angeli, 2005; Lahtinen & Pehkonen, 2013). However, there is evidence 
that not all university students reach the most highly developed level of personal epistemology (Kuhn & 
Weinstock, 2002; Kaartinen-Koutaniemi & Lindblom-Ylanne, 2012; King & Kitchener, 2004; Perry, 1970). 
King and Kitchener (2004) have found that only advanced doctoral students consistently show the highest 
level of epistemological beliefs. Furthermore, Kaartinen-Koutaniemi and Lindblom-Ylanne (2008, 2012) 
have shown that there is a considerable variation in personal epistemology among final-year master’s 
students. Their results also showed variations between students in different age groups, study phases and 
disciplines (see also Hofer, 2006; Muis, Bendixen & Haerle, 2006). In addition, researchers have assumed 
that students’ epistemological beliefs may vary within the same discipline or domain (Hammer & Elby, 
2003; Greene & Yu, 2014). 


5. Critical thinking and different conceptions of knowledge 

As the brief review above indicates, the literature of personal epistemology makes a distinction 
between a lower level of epistemological beliefs, in which knowledge is perceived as consisting of 
unchanging facts and is acquired directly from external authorities, and higher level epistemological beliefs, 
in which knowledge is seen as uncertain and constructed by the individual himself (Kuhn & Weinstock, 
2002; Hofer, 2005; Valanides & Angeli, 2005). Several researchers have stated that students with higher- 
level epistemological beliefs have better critical thinking skills than students with lower level 
epistemological beliefs (King & Kitchener, 2004; Kuhn & Weinstock, 2002; Kuhn, 1999; 2005). Recently, 
Holma and Hyytinen (2014) have argued that there are several conceptual problems in this kind of 
hierarchical theory of knowledge (see also Elby & Hammer, 2001). In this section we focus on three 
conceptions of knowledge identified in the review of the literature on epistemology. These conceptions, 
specifically relativism, metaphysical realism and fallibilism, have theoretical importance for conceptualising 
critical thinking. 

A relativist position implies that all knowledge is relative to the person who believes or that all 
interpretations, theories and beliefs are equally right. Because all beliefs are equally right, there is no reason 
to compare and evaluate different beliefs—all beliefs are equally justified (Holma, 2012; Holma & Hyytinen, 
2014). The problem of relativism becomes clear when it is related to the concept of critical thinking (Holma 
& Hyytinen, 2014). Given that relativism allows people to construct their own ‘personal truths’, critical 
thinking turns out to be unnecessary (Bleazby, 2011). For example, there is no need to evaluate ideas or 
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search for alternatives, because all ideas are equally trustworthy and justifiable (Bleazby, 2011; Holma & 
Hyytinen, 2014). Therefore, the idea that critical thinking presupposes the relativist view of knowledge is 
untenable. 

Metaphysical realism is an epistemological position that assumes that ‘our knowledge and symbol 
systems [i.e. theories] directly reflect the structure of reality’ (Holma, 2004, p. 421; Putnam, 1981). The 
literature of personal epistemology seems to understand realism as metaphysical realism (see e.g. Kuhn 
2005; Kuhn & Weinstock, 2002; see also Holma & Hyytinen 2014), and furthermore, it appears to connect 
with metaphysical realism the assumption of the possibility of the certainty of human knowledge. As King 
and Kitchener (2004) put it, knowledge is ‘obtained with certainty by direct observation’ (p. 7). 1 In the 
context of metaphysical realism, critical thinking turns out to be pointless. 

Fallibilism is an epistemological position that implies that all our beliefs are liable to error (Reed, 
2002; Niiniluoto, 1999; Holma, 2012). Contrary to relativism, fallibilism does not assume that all beliefs or 
theories are equally right. It presumes the possibility of improving our current conceptions, theories or 
beliefs. As Holma (2012, p. 399) aptly states of fallibilism, ‘this position, like the belief that all human 
knowledge is uncertain, coheres with the evolutionary understanding of knowledge: the bodies of knowledge 
we now have may be mistaken and thus [are] possible subjects for revision, but they have, nevertheless, 
survived the process of evolution to this point; as such, they provide the best available starting point for 
choices and action of the present moment concerning further inquiry’ (see also Peirce, 1934). From this point 
of view, epistemological fallibilism fits the presumption of critical thinking. Previous research on personal 
epistemology lacks the notion of epistemological fallibilism. 


King and Kitchener (2004) do not call the lowest level of reflective thinking realism. However, in their 
model they maintain that, at the most limited level of thinking, knowledge is certain and is obtained from 
direct observation (p.7). This position fits metaphysical realism. 
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Table 1. 

Summary of the key concepts of this study 


Concept 

Description 

Critical thinking 

Process that enables an individual to make an informed 
decision between conflicting claims. It involves skills and 
dispositions (e.g. attitude and motivation) to evaluate the 
reliability and relevance of evidence, to identify 
arguments, to analyse, interpret and synthesise data from 
a variety of sources, to draw valid conclusions and 
address opposing viewpoints). 1 Critical thinking also 
involves ‘knowing how to do something’ (procedural 
knowledge) and ‘knowing that’ (propositional 
knowledge). 2 


Students’ thoughts/beliefs about the nature of knowledge 
and the nature of knowing, including personal beliefs 
about themselves as knowers. 3 

The idea that human beliefs are direct copies of reality. 

The belief that all human knowledge is certain is 
connected to this epistemological position. 4 

The view that all knowledge is relative to the person who 
believes or that all interpretations/beliefs are equally 
correct. Because all beliefs are equally correct, there are 
no means for comparing different beliefs. 5 

The view that human knowledge is uncertain. In contrast 
to relativism, it presumes the possibility of improving our 
current conceptions, theories or beliefs, seeking criteria 
for evaluating, comparing and justifying these beliefs or 
theories. 5 

'Based on Bailin & Siegel (2003); Ennis (1991); Fisher (2011); Fisher & Scriven (1997), Siegel (1988). 

2 Based on Scheffler (1965); cf. also Ryle (1949). 

3 Based on Pintrich (2002). 

4 Based on Holma (2004); Putnam (1981). 

5 Based on Holma (2012); Holma & Hyytinen (2014); Peirce (1934). 

Table 1 provides a summary of the definitions of the key concepts in this study. With this broader 
framework we are able to pin down different areas in critical thinking and epistemological beliefs, which 
have been shown to be vital for conceptualising these phenomena in prior studies or theorizations. Although 
the conventions of critical thinking and epistemological beliefs are commonly embodied in social practices 
(e.g. Arum & Roksa, 201 lb; Elby & Hammer, 2001; Kuhn, 2005), the underlying dimensions (i.e. evaluating 
the reliability and relevance of evidence, identifying arguments, analysing information, addressing opposing 
viewpoints, reasoning) are relevant in each scientific discipline. Moreover, in line with previous studies we 
expected that students’ epistemological beliefs and critical thinking might vary within the same discipline 
(see Greene & Yu, 2014; see also Bailin & Siegel, 2003). 

In our study we focused on the qualitative differences in critical thinking and personal 
epistemological beliefs by examining ten third-year university students’ thinking and performance in a 
cognitively-demanding authentic problem-solving situation. The aims of this study are twofold: to identify 
and describe qualitative differences in third-year university students’ critical thinking skills and 
epistemological beliefs in a problem-solving situation, and to analyse the interconnections between students’ 


Epistemological beliefs 


- metaphysical realism 


- relativism 


- epistemological fallibilism 
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personal epistemologies and critical thinking skills. To achieve these aims, we formulated the following 
research questions: (1) How are critical thinking and epistemological beliefs presented in a problem-solving 
situation in a specific group of third-year university students? (2) How do critical thinking and 
epistemological beliefs vary from one individual to the next? 


6. Research methods and materials 


6.1 Participants 

This study was conducted with ten third-year bioscience students drawn from the fields of biological 
and environmental sciences in a research-intensive university in Finland. The target population consisted of 
all third-year bioscience students in this particular university. First, we selected 40 students at random 
(approximately one-half of the target population). Then we invited all students selected to participate in our 
study. Ten out of 40 students volunteered. Seven of the participants were female and three male. The 
students’ ages varied from 22 to 29, the mean age being 24. All came from a homogeneous cultural 
background, and all shared the same first language (Finnish). In addition, the students had the same national 
high school certificate and had enrolled in the same bachelor’s study programme. The participants were at 
the same phase of their studies, that is, near the end of their bachelor’s studies, with the exception of one 
student whose study pace had been slower. During their university careers, the students had participated in 
lectures, practical laboratories, seminars, field courses and web-based teaching. We are aware that the 
sample size is too small for generalization. However, the purpose of this study is to deepen understanding of 
critical thinking and epistemological beliefs, for example, so as to describe how these phenomena vary 
across individuals in this specific group of students. 


6.2 Procedures 

For this study we collected a large body of data for each participant using a multi-method approach 
(Johnson, Onwuegbuzie & Turner, 2007), including think-aloud protocol, interviews and a Collegiate 
Learning Assessment (CLA) performance task. The data collection was carried out in the spring of 2010 and 
consisted of ten cognitive labs. The students came to a classroom and were given the details of the study. 
The students spent two to three hours reading and responding to the performance task. In responding to the 
task, the students were asked to verbalise their thoughts (to ‘think aloud’). In the course of carrying out the 
task while thinking aloud, the students were also asked to write a memorandum addressing critical issues in 
the task and recommending —and justifying— a course of action. Following the task, the students were 
interviewed about their processes in carrying out the task. Students were also asked questions about critical 
thinking, knowledge and knowing. Details of the procedures are provided below in appropriate sections. 

6.2.1 Collegiate Learning Assessment (CLA) 

The Collegiate Learning Assessment (CLA) instrument for assessing college-level critical thinking 
skills used in this study was developed by the Council for Aid to Education (CAE). The CLA is a 
standardised, open-ended test and it measures analytical reasoning, problem solving and written 
communication. Unlike most standardised tests used in measuring critical thinking, the version of the CLA 
used here did not include any multiple-choice questions (Klein, Benjamin, Shavelson & Bolus, 2007). The 
CLA consists of two elements: a set of performance tasks and a set of analytical writing tasks (Shavelson, 
2010). Only the performance task was used in this study. Recent studies have found that open-ended 
problems with no obvious solution provide an opportunity for students to reflect on their beliefs about 
knowledge (Barzilai & Zohar, 2012; Ferguson & Braten, 2012). For example, in a problem-solving situation 
students would need to determine the trustworthiness, and relevance, of different types of information 
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presented to them, co-ordinate various pieces of information related to the problem and consider the 
underlying assumptions and claims (Shavelson, 2010). 

The CLA performance task presents a realistic situation or problem and includes directions, open- 
ended questions and a document library containing reading material. In order to respond to the task, the 
students need to read, organise, synthesise and analyse information (which might be reliable/unreliable; 
relevant/irrelevant to the completion of the task; see Shavelson, 2010) from multiple documents (for example 
letters, memos, summaries of research reports, articles, diagrams, graphs, maps, interview notes). In doing 
these activities the students need to assess their confidence in information taken from various sources, 
including the relevance of the source, and thereby deal with conflicting information. They then need to 
decide on a course of action and provide a reasoned explanation and justification for their course, drawing on 
supporting information from the document library (Klein et al., 2007; Shavelson, 2010). They also have to 
argue for and against alternative explanations. The specific performance task used in this study is proprietary 
and consequently cannot be described here. An example of a representative CLA performance is presented in 
Figure 1. 


DynaTech Performance Task 

You are the assistant to Pat Williams, the president of DynaTach, a company that 
makes precision electronic instruments and navigational equipment. Sally Evans, a 
member of DynaTech's sales force, recommended that DynaTech buy a small 
private plane (a SwiftAir 235) that she and other members of the sales force could 
use to visit customers. Pat was about to approve the purchase when there was an 
accident involving a SwiftAir 235. You are provided with the following 
documentation: 


1: Newspaper articles about the accident 

2: Federal Accident Report on in-flight breakups in single engine planes 
3: Pat's e-mail to you and Sally's e-mail to Pat 
4: Charts on SwiftAir’s performance characteristics 
5: Amateur Pilot article comparing SwiftAir 235 to similar planes 
6: Pictures and description of SwiftAir Models 180 and 235 


Please prepare a memo that addresses several questions, including what data 
support or refute the claim that the type of wing on the SwiftAir 235 leads to 
more in-flights breakups, what other factors might have contributed to the 
accident and should be taken into account, and your overall recommendation 
about whether or not DynaTech should purchase the plane. 


Adapted from R. Shavelson, 2010, Measuring College Learning Responsibly: Accountability in a 
New Era. Stanford, CA: Stanford University Press, p. 38. 

Figure 1. An example of a CLA performance task. 

6.2.2 Cognitive labs 

The purpose of cognitive labs is to study the cognitive processes that students use when they 
complete different tasks. Students are asked to report their thoughts verbally as they carry out a task (see 
Johnstone, Bottsford-Miller & Thompson, 2006). In this study cognitive labs were divided into three parts: 
(1) instruction and training, where the researcher explained what the cognitive lab was about and trained the 
students to think aloud with a short warm-up task; (2) ‘think-aloud’, where the students talked aloud while 
completing the CLA performance task; and (3) a follow-up interview. The cognitive lab for each student was 
video-recorded and lasted two to three hours. To ensure the consistency of cognitive labs, a script of 
directions and the same training task and the interview questions for each student were used. The videos 
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were recorded with two cameras and a table microphone. The cognitive workshop produced the following 
materials: video data, content logs (see below), written test answers and transcribed interview data. 

The neutral type of think-aloud protocol conducted by Ericsson and Simon (1993) in which students 
were not interrupted while they were performing a task was used in this study. The think-aloud method 
makes it possible to collect data about a student’s ongoing thinking processes whilst he or she is working on 
a task (Ericsson & Simon, 1993; Cotton & Gresty, 2006; van Someren, Barnard & Sandberg, 1994). We 
assume that students’ ‘knowing-that’ attributions (e.g. ‘scientific knowledge is true’) may reflect their 
epistemological orientations and reveal their criteria for justifying beliefs (see Scheffler, 1965). Moreover, in 
some cases the think-aloud method makes it possible to explore critical thinking in action, especially in 
situations that simulate real-world circumstances. 

Immediately after the task was performed, a follow-up interview was conducted. The aim of the 
interview was to gain more detailed information about the processes and knowledge that the students used to 
complete the task and to probe students’ beliefs about knowledge and knowing. For example, the students 
were asked questions about how they dealt with conflicting information, how they decided which 
information to use, what sources of information in documents from the documents library they trusted and 
why, and how they usually evaluate knowledge. 


7. Data analysis 

The data were analysed using a qualitative thematic analysis with an abductive approach 
(Timmermans & Tavory, 2012; Haig 2005). An abductive strategy means that the themes identified from the 
data were linked to the theoretical understanding based on previous studies. Abduction is a process that 
combines things which one had not previously associated by creating a new interpretation, that is, the 
relationship of a new combination of study features (Timmermans & Tavory, 2012). Hence, the analysis 
process was nonlinear, moving back and forward amongst all the data, data items, analysed qualities and 
understanding of the phenomenon based on prior studies. The first and fifth authors were responsible for the 
analysis, but the final results were obtained through a thorough discussion with all authors. The data were 
processed in such a way that the participants could not be identified. 

The analysis included four phases (Figure 2) that represented the unique combination of data- 
grounded and theory-driven phases, as well as phenomenon and individual-level analyses. During the first 
phase, video recordings were initially indexed with the ELAN program, which allows the addition of as 
many tiers and annotations on the video stream as needed (see Lausberg & Sloetjes, 2009; Max Planck 
Institute for Psycholinguistics, 2012). The purpose of indexing was to make the large video data set easier to 
handle. In this study the indexing tiers corresponded to the parts of cognitive labs including training, think- 
aloud methods and interviews. In addition, students’ interviews from the videos were transcribed. 
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ANALYSIS 


1. Phase: Familiarizing oneself with data, data grounded phase 

of Watching video data, b) initial indexing of video material* with ELAN program, 
c) Transcribing interviews* d) Generating content log s, e) Reading and re-reoding the data, 
f) Noting down initial ideas far coding. 


2. Phase: Coding, theory-driven phase ▼ 

a) Generating coding features, 

b) Coding relevant features systematical across the entire data set and within the data items such 
as transcribed verbal protocols and written answer from one person. 


4. Phase: Generating 
student profiles, focusing 
variation on an individual 
level 

a) Defining student profiles. 
Focusing on the data set of 
one student. 

6,1 Cross-checking if the 
profiles work in the entire 
data set 



l. Phase: Thematic analysis*, 
variation at a phenomenon level 
a) Searching for themes: Collating 
codes into potential themes, 
gathering all relevant data far each 
potential theme. Focusing the 
entire data set. 

bj Defining, naming, and reviewing 
themes: Ongoing analysis to refine 
the specifics of each theme, 
generating clear definitions and 
names for each theme. Cross* 
checking if the themes work in the 
entire data set. 


RCSULTS 


2 

mi 

< 
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Theme 1 

Fleubilrty in critxaJ thinking 


Subtheme 1 

Flexibility 


Theme 2 

Variation critical thinking and eptstemoJogicai beliefs 




Subtheme 1 


Subtheme 2 

Subtheme 2 


Critical thinking as. a 


Critical thinking as a 

Rigidity 


tool for enhancing 


tool for determining 



l understanding 


truth or falsehood 




Profile 1 


Profile 2 


Thorough 


Superficial 


processing 


processing 




i_y 



> 

s 


*Based on Braun & Clarke (2006, p. 87). 

Figure 2. A visualisation of the analysis process. 


After the indexing, content logs were created for each video in which accurate descriptions and 
summaries of events were systematically recorded. Transcriptions of relevant sections of verbalisations of 
students’ critical thinking and epistemological beliefs (e.g. whenever a student evaluated the quality and 
reliability of the information in a document or where a student reached a conclusion based on her or his 
analysis) and nonverbal acts (e.g. a student did not read in detail or skipped over the document) were also 
included in the log (cf. Table 1). 

The second phase of the analysis was the data coding (see Table 2 for definitions). This phase was 
theory-driven, meaning that the features guiding the coding were based on prior studies (see Table 1). The 
coding focused on the following qualities: the process by which the student approached the task and solved 
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the problem, the knowledge that the student used to carry out the task, the critical thinking exhibited, and 
epistemological beliefs. These different qualities were coded systematically across the entire data set and 
within the data items such as the transcribed interviews and the think-aloud videos of each person. By this 
means, all the data items from one student, including the video data, content log, written test answers and 
transcribed interviews, were coded and analysed separately, after which data from all students were 
combined and compared (see Table 3 for an example of the codes). All extracts were labelled with a student 
code (S1-S10) and a method code (1= interview, T=think aloud, W= written test answer). The data examples 
were translated into English. 

Table 2. 

Data sources and focal points of coding 


Data Sources 

Coding Features 

Video data, content logs, transcribed interviews 

1. The process: how does the student approach the 
task and solve the problem? 

Video data, students’ written answers, content logs, 
transcribed interviews 

2. What knowledge/information does the student use 
to solve the task? 


2.1 What kind of knowledge/information did the 
student use? 

2.2 Why? 

2.3 How does the student use that 
knowledge/information? 


Video data, students’ written answers, content logs, 3. Critical thinking 
transcribed interviews 

3.1 How does the student identify, analyse and 
evaluate information, ideas and arguments? 

3.2 How does the student judge the acceptability 
(especially the credibility) of documents? 

3.3 How does the student interpret data/ graphs/ 
maps? 

3.4 How does the student recognise the relationship 
between assumptions? 

3.5 How does the student evaluate background 
information? 

3.6 How does the student make a decision ? 

3.7 How does the student identify reasons and come 
to a conclusion? 

3.8 How does the student produce explanations and 
arguments? 
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Video data, students’ written answers, content logs, 4. Epistemological beliefs 
transcribed interviews 

4.1 What does the student think about knowledge, 
knowing and the credibility of knowledge? 

4.2 How does the student determine the 
trustworthiness, acceptability and justification of 
different types of information? 

4.3 How does the student describe herself or himself 
as a knower? 


Table 3. 


An example of codes 


Data Extract 

Coded for 

You could consider this a good argument; the 
expert has gone [to the place where events took 
place] to see for himself (S9T) 

4.1 What does the student think about 
knowledge, knowing and the credibility of 
knowledge? 

4.2 How does the student determine the 
trustworthiness, acceptability and justification of 
the different types of information? 

- - yeah, I don’t believe [the chair of the 
stakeholder group] is completely off the mark 
either. [Reliability] is just always case-specific. 
(SSI) 

4.1 What does the student think about 
knowledge, knowing and the credibility of 
knowledge? 

This just seems scientific somehow. (S6T) 

4.1 What does the student think about 
knowledge, knowing and the credibility of 
knowledge? 


In the third phase the codes and coded extracts were grouped under potential themes, and all the 
relevant data were gathered under each theme (Braun & Clarke, 2006). We identified a variety of 
preliminary themes on the basis of the codes. During the analysis, the preliminary themes were defined and 
combined several times. In the end two main themes and two subthemes remained (see Figure 2). The final 
themes were refined, labelled and cross-checked to see if they worked in relation to the coded extracts and 
the entire data set. The focus of the thematic analysis was the variation of study features on the phenomenon 
level. 

After completing the thematic analysis, we found that the students could be placed in different 
profiles based on our themes as well as on patterns of behaviour and cognition observed. This phase focused 
on the variation of study features at the individual level. Thereafter, we conducted final descriptions, 
interpretations and revisions of the results. The results of thematic analysis show how critical thinking and 
epistemological beliefs manifested themselves in this particular group of students, whereas the student 
profiles describe how these phenomena vary across individuals. 


8. Results 

In the thematic analysis two main themes were identified: (1) flexibility in critical thinking and (2) 
variation in critical thinking and epistemological beliefs. The two themes emerged from exploring the 
students’ critical thinking from different perspectives. The ways in which the themes were related differed 
amongst the participants, which further allowed us to identify student profiles. We identified two main 
profiles, and on the basis of their characteristic features we labelled them as (1) thorough processing and (2) 
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superficial processing. The results are described using a combination of identified themes and student 
profiles. 


8.1 Flexibility in critical thinking 

Students showed various skills in their ability to adapt their thinking and their performance 
flexibility to the demands of the task. There was clear variation in the students’ ability to change their actions 
or ways of critical thinking, in which we identified both rigidity and flexibility. Flexibility meant that the 
students could modify their actions and processes and change their behaviours as needed, whereas rigidity 
refers to situations in which students could not change their processes or look at things from a new 
perspective or adjust to new evidence in a problem-solving situation. Students who were able to make 
changes in their actions showed open-mindedness and an inquiring attitude. 

In the following extract, one student describes how he adjusted his performance and ended up 
analysing and interpreting the documents correctly: 

I approached [this assignment] maybe a little too much as if I had simply copied what they say here in 
these papers and put them down [in my answer]. But then when I started thinking, like about my own 
views on the topics, then right off in [question] number one, it took me a really long time to answer this 
question. (S8I) 

On the other hand, there were students who could not adjust their thinking or performance. Some of 
these students said that they always act in the same way: 

Well, I’m always like this time-management catastrophe. Like in exams and everything, especially 
exams, it always feels like I run out of time. And in general I notice that in all comprehension and 
analysis assignments and things like that, they always take me a really long time. (S5I) 


8.2 Variation in critical thinking and epistemological beliefs 

Students showed various aims in the problem-solving situation. Some students tried to understand 
the complex situation, whereas others tried to find the right answer to the problem. Students also varied in 
their critical thinking, including (a) their disposition and ability to identify, analyse, evaluate and interpret 
information; (b) their ideas and arguments in judging the acceptability of documents; (c) their abilities to 
recognise relationships between assumptions; (d) their abilities to make a reasoned decision; and (e) their 
abilities to produce explanations and arguments. In addition, students’ epistemological beliefs varied. Some 
students claimed that only through scientific knowledge we can arrive at truth. However, other students 
expressed the idea that both objective and subjective knowledge can hold the highest epistemic status. 

We found that critical thinking emerged as a tool for understanding knowledge and determining the 
goodness and reliability of knowledge; thus, students’ epistemological beliefs were interwoven into their 
critical thinking. Within this theme we found that students used critical thinking either a) as a tool for 
enhancing understanding or b) as a tool for determining truth or falsehood. Based on this difference, students 
could be classified in one of two qualitative student profiles, either (1) thorough processing or (2) superficial 
processing. The profiles captured the diversity of the students’ abilities and dispositions to think critically. In 
addition, these two profiles characterised the variation in how students viewed the nature and limitations of 
knowledge and knowing, and especially in how they determined what is needed to evaluate knowledge as 
true or justified and how they acquired and used the knowledge in the problem-solving situation (see Table 
4). The phrase ‘acquiring knowledge’ here emphasises the dominant way that students used to obtain 
knowledge in a problem-solving situation. 

The students classified in the profile called ‘thorough processing’ demonstrated an ability to carry 
out a deep processing of the content of the documents. These students saw knowledge as fallible and 
contextual. Similarly, the students in the profile called ‘superficial processing’ expressed the idea that 
knowledge is fallible, yet they did not consider the contextual nature of knowledge at all. In the problem- 
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solving situation they did make a serious effort to analyse, interpret or synthesise the information in the 
materials. The thorough processing profile is further divided into two sub-profiles: (1A) reasoning in order to 
reach conclusions and (IB) intuition. Likewise, the second profile, ‘Superficial processing’, also consisted of 
two sub-profiles: (2A) referring to an argument made by authoritative specialists or experts and (2B) trust in 
scientific method and proof. We describe the characteristics of the profiles and sub-profiles below and 
provide details pertaining to variation in academic thinking. 

Table 4. 

The nature of knowledge and acquiring knowledge in two qualitatively different student profiles of critical 
thinking 


Sub-theme 

Student profile 

Epistemological beliefs 

Acquiring knowledge 
(sub-profile) 

Critical thinking as a 
tool for enhancing 

Thorough 

Both objective and 
subjective knowledge 
can hold the highest 

Reasoning in order to 
reach a conclusion 

understanding 

processing 

epistemic status. 
Knowledge is fallible, 
relative and contextual. 

Intuition 

Critical thinking as a 
tool for determining 
truth or falsehood 

Superficial 

processing 

Knowledge may reach 
truth only if it is 
produced by a reliable 
process, that is, using 
empirical methods. 
Objective knowledge 
holds the highest 
possible epistemic 
status, but is fallible. 
Some theories may be 
false. 

Referring to an 
argument made by 
authoritative 
specialists/experts 

Trust in scientific 
method and proof 


8.3 Profile 1: Thorough processing students 

The students (n=5) who deeply analysed the content of the documents created their own 
understanding of the problem-solving situation. For them, critical-thinking skills were tools to deepen and 
enhance understanding. These students believed that theories and beliefs could be understood in relation to 
some context, as the following extract shows: 

- - yeah, 1 don’t believe [the chair of the stakeholder group] is completely off the mark either. [The 
reliability of] knowledge is just always context-specific. (S8I) 

These students considered it possible to improve current theories and beliefs. These students were 
thus open to new evidence that could disprove a previously-held position or belief. For them, scientific 
knowledge is probably reliable. They believed that both objective and subjective knowledge could attain the 
highest epistemic status, meaning that subjective perceptions (e.g. their own perceptions or information 
obtained from someone else) could also be reliable. These students thought that the credibility of knowledge 
could be affected by vested interests or bias, for example. Although these students emphasised their own role 
in constructing knowledge, they did not believe that all knowledge is constructed or generated by human 
minds. From the epistemological perspective, these students took the fallibilist position. 
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The students who belonged to the ‘thorough processing’ profde were further divided into two sub- 
profiles on the basis of how they acquired knowledge and reached conclusions in the problem-solving 
situation and how flexible they were in changing their actions or ways of thinking. The first sub-profile was 
called (1A) reasoning in order to reach conclusions and the second was called (IB) intuition. 

8.3.1 Reasoning in order to reach conclusions 

Two students endeavoured to reach conclusions by reasoning. These students analysed connections 
across the information presented in the different documents. They also clarified and interpreted different 
claims and ideas that were presented in the documents. On the basis of their own analyses, they synthesised 
information, reached a clear decision or conclusion, provided arguments for their decision and explained 
why this decision was the best in light of all the issues brought up in the documents. In the following 
example one student describes her analytical process: ‘Somehow I kn ew how to read beyond [the 
documents]’ (S4I). These students were also able to adjust their thinking in line with new evidence and make 
changes in their actions. These students justified conclusions with good reasons (e.g. reliable and valid 
evidence) and considered themselves as active and responsible knowers, as the following extracts show: 

But maybe I wouldn't, like, start criticising right away; somehow, I’d have to start looking into, you 
know, on what basis they arrived at these figures. (S101) 

For instance, using this graph is fine, but 1 think it’s been, you know, clearly misinterpreted here in the 
text. (S4I) 

These students created their own understandings of the situation on the basis of their analyses. They 
used the materials for the analysis and evaluation process in a way that went beyond the obvious. For 
example, they identified, analysed, evaluated and interpreted all the major facts and ideas presented in the 
documents. They consciously excluded some information in the documents because of contradictory 
evidence. In addition, they were able to distinguish relevant claims from irrelevant ones. These students also 
judged the reliability of the documents, evaluated presuppositions and analysed connections between claims. 
Furthermore, they produced different explanations, identified reasons, produced arguments and drew 
inferences. 

These students further identified and used several criteria in evaluating reliability: corroborating 
claims from different sources, evaluating the context in which the claim was made, exploring who 
interpreted the data and evaluating the presuppositions. Moreover, these students considered the ethical 
aspects of knowledge: knowledge and information shape human beings’ worldview. 

I’ve just gotten the impression about newspapers, about the media too, that it somehow has the effect 
that the opinions [presented in them] are so strong that maybe you don’t analyse it so clearly. So like 
even Helsingin Sanomat [Finland’s largest daily] really has, somehow it seems that they have a pretty 
strong, you know, bias... you know that even if it’s neutral in a way, then the fact the issues they raise in 
it, in a way that already affects what information is raised, and what... that it like really powerfully 
shapes people’s worldview. (S4I) 

8.3.2 Intuition 

Three students justified conclusions by intuition. These students created their own understanding of a 
situation. However, they did not select materials or question any information: they used all the information in 
the documents, such as empirical knowledge, expert opinions, reports, maps, experiences of an inhabitant, 
recommendations, letters and second-hand knowledge. These students acquired knowledge in a rather 
uncritical way. They rarely evaluated the reliability of documents. Indeed, these students did not have clear 
criteria for evaluating the reliability or relevance of information. They just trusted their intuition: 

This just seems scientific somehow. (S6T) 
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I don’t know how 1 should formulate this, but I ’ll start by saying that when / read, for instance... or 
when I ’m taking classes, I don’t spend a whole lot of time wondering if some piece of information is 
reliable or not. (S6I) 


These students started to analyse and interpret thoroughly all information presented in the 
documents. They identified all major facts and ideas. They also considered different decisions or 
explanations, but could not explain what decision was the best or why. There were too many options 
available. Because the students did not reach clear conclusions, they did not present any arguments for 
accepting the conclusion either. These students showed an inability to adjust their thinking to new evidence 
or make changes in their actions. 

8.4 Profile 2: Superficial processing students 

Common to all students in the second main profile was that they processed the materials in the 
problem-solving task superficially: they did not make a serious effort to analyse, interpret or synthesise the 
information in the materials. This profile consisted of five students who used critical thinking as an 
instrument for determining truth or falsehood. Their goal was to find the right answer to the problem. In 
contrast to the ‘thorough processing’ profile, students in this profile believed that knowledge is trustworthy 
only if it was produced through a reliable process, for example, by using empirical methods or consulting 
suitable experts. For these students, scientific and verified knowledge is the most reliable, because that kind 
of knowledge is based on evidence, and it is unbiased and objective. The students believed that subjective 
knowledge is predominantly untrustworthy. However, these students considered empirical knowledge (which 
holds the highest epistemic status) to be fallible too, not absolutely certain. They believed that some theories 
might be false and that it is possible to improve current conceptions and theories. From the epistemological 
perspective, these students also took the fallibilist position. 

The analysis indicated varying problems in critical thinking, such as problems in evaluating 
information, reasoning and reaching conclusions. Some of these students also had little motivation to think 
critically. Characteristic of the students in this profile was that they focused on isolated details. They took 
knowledge for granted. In other words, they accepted knowledge (particularly scientific knowledge) as true 
without question. These students were further divided into two sub-profiles according to how they acquired 
knowledge in a problem-solving situation, trusting either (2A) an argument by authoritative specialists or 
experts and (2B) verified empirical evidence or testimony. 

8.4.1 Referring to an argument by authoritative specialists or experts 

Two students were categorised in this sub-profile. These students trusted authorities in acquiring 
knowledge. They saw themselves as uncertain knowers. These students believed that if a person who is said 
to be an authority on something makes an argument about that something, then the argument should be 
trustworthy and therefore usable. The right answers can be reached by consulting the right expert. These 
students repeated arguments and conclusions as these were presented in the documents. They drew on 
empirical knowledge and expert opinions, that is, arguments from authoritative sources. 

These students had difficulties in evaluating information. They focused on details and took in all the 
information they were presented without question. They picked up isolated and obvious details from the 
materials for each question. The students did not properly analyse, evaluate or interpret the information 
presented in the documents; they just jumped to conclusions. They disregarded and seriously misinterpreted 
important information. They also had problems in reasoning and reaching a conclusion. In order to make 
decisions or arguments, these students reproduced lists of isolated details from documents. They did not 
provide any reasons or explanations for their decisions. Moreover, they did not identify alternative solutions. 
These students presented some unreliable claims as being credible. In the interview one student representing 
this sub-profile said that she has had similar problems in learning: 
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Creative comprehension and, like, reaching a syn thesis of overall concepts is really challenging for me. 
Like, for instance, it’s really hard to study for exams, because I’d be more than happy to read the book, 
but then I don ’t really grasp the key message and structure that it’s trying to communicate. Acquiring 
data independently and, like, learning information that way is challenging. So, for instance, I haven’t 
done all my exams. 1 haven’t done them because, I’ve tried to start [studying for] them lots of times, but 
then some, how would you put it, if listening is auditory, then learning from text is pretty hard for me. 
This third year, which is currently underway, has been, like, really hard. I’ve really haven ’t gotten 
many credits. I don’t feel I’ve accumulated the amount of information I should have or could have in 
three years. That the pieces of information are discrete and still pretty scattered in my head at the 
moment. (S5I) 

These students expressed the view that knowledge is always uncertain, but they did not consider 
themselves capable of evaluating knowledge. These students named a few external criteria for evaluating 
knowledge (such as an authority, expert opinion, publication, openness, journal citations). However, in 
practice they did not know how to use these criteria independently. Both of these students gave authoritative 
experts the responsibility for evaluating knowledge, as the following example demonstrates: 

I don’t know what the right approach is in order to grasp those overall concepts from that huge mass of 
teensy-weensy details. Because a candidate has to read a huge number of articles to find the ones that 
are, like, related to one’s own topic and all. So it’s really hard when you ’re, like, reading an article to 
judge why this one might be better than that one. So. But I got a tip from my supervisor that I should 
pay attention to the reliability of the journal. To be honest, it’s the research articles, the ones we have at 
the university, that are actually the only ones we’re told we can cite. And then it’s like... they’re easy to 
evaluate based on which publications are more credible. And on the Web on \sic] Science, they have 
this one like... what is it, like an indicator that they have, just based on the number of citations and 
other factors, of the accuracy of the research data... It’s hard! In a way, to make that distinction 
between what’s true and what isn’t. At least I don’t have the know-how to say what’s true and what 
isn ’t. (S5I) 

Both students expressed the view that in a real-life situation they would seek help from other people, 
such as authoritative specialists (e.g. a university teacher) or other students. For example, one student 
representing this sub-profile said several times that she needed co-operation with other students to solve the 
task: 


I haven ’t really had to do anything like this before. That it’s pretty hard in a way. There are so many 
points and, you know, perspectives here. I haven ’t even had to think about stuff like this at the 
university, then it’s really like new for me, or you know. The assignment was pretty difficult. This might 
have been more interesting as a group assignment. Like there would have been, you know, interaction, 
and then maybe it would have generated more thoughts somehow. (S7I) 

8.4.2 Trust in scientific method and proof 

The three students comprising this sub-profile were very critical. They all selected documents 
roughly based on empirical evidence, excluding more than half of the documents provided. These students 
were aware of their own behaviour: 

I eliminate some of the documents right away, for instance, email exchanges and letters, because the 
people haven’t investigated the matter; the text was written based on a gut feeling. (S3T) 


These students expressed the view that scientifically and empirically verified knowledge is the most 
reliable. They knew that corroboration from other reliable and related sources improves credibility. In the 
problem-solving situation they only trusted and used arguments by scientific authorities. These students 
described themselves as ‘error seekers’ in the interviews. The following examples illustrate the view of the 
students in this sub-group: 
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I trust exam books and articles a lot, yes. The difference between the two is that books can often, you 
know, be unreliable. Plus the fact that, at least when they ’re academic, it has a lot to do with when they 
were written, because things move so fast. That I have this one book for my thesis that I was just looking 
at, it's got tons of mistakes. So, like, you just have to find them yourself. But with articles, probably 
those, and then of course depending on the journal. That maybe some article in Science: I consider 
them pretty reliable. Nowadays, I’m a little too sceptical about all kinds of things. 1 question a lot more 
these days than I used to. (SII) 

You can get the first impression [of reliability], of course, from the kind of source it was published in. In 
other words, I wouldn ’t swallow some Iltalehti [a Finnish tabloid] headline on some scientific subject 
without thinking it over properly first. But having a reference to those academic publications, and as far 
as how I’ve drawn those conclusions myself after having read the article, not based on some newspaper 
headline, then that would be at least important in terms offirst impressions. And... well, even if you 
read a scientific article, if it doesn ’t agree at all with what you’ve learned about the topic earlier, then 
of course you ’d have good cause to suspect those research results quite a bit. But the source is what I’d 
probably consider as the main thing. (S3I) 


Although these students describe themselves as critical, they did not evaluate information from 
reliable sources in the problem-solving situation. For example, they did not recognise that two sources, 
which included empirical or verified knowledge, were biased. They analysed and interpreted information 
superficially and focused on isolated details. They did not interpret the documents they selected nor did they 
consider presuppositions. In order to draw conclusions these students mainly reproduced details from the 
documents. They did not identify alternative solutions or conclusions or approaches to the problem. Nor did 
they provide any reasons or explanations for their own conclusions. They identified only a few claims that 
were presented in the documents and disregarded many relevant aspects of those claims. As a result they had 
problems reaching a conclusion. In the following example one student described the situation as follows: ‘At 
least I wouldn’t draw any conclusions based on those [documents]’ (SIT). All these students thought that 
there was one definite answer to the problem. 

One student in this sub-profile emphasised that she does not have any disposition or motivation to 
express reasons for or against some idea in a test situation or in everyday life: 

In everyday life it’s rare that, if you ’re discussing something, it’s rare that anything like this happens. 

Or I never, really rarely discuss anything argumentatively in any way. In real life I simply don’t like it, 
discussing issues. (S9I) 


8.5 Summary of the results 


Figure 3 combines the two main themes in order to form a comprehensive picture of participants’ 
critical thinking. Students who had several problems in critical thinking, yet had flexibility coped with the 
demands of the task. For example, two students had problems evaluating documents and did not form a 
general picture of the situation presented in the documents. Because the students were struggling with the 
demands of the task, they selected documents and reproduced arguments and conclusions just as these were 
presented in the materials. Eventually, the students reached a limited conclusion. On the other hand, there 
were students who were skilled in specific critical-thinking skills, such as analysing and interpreting 
information, but lacked other abilities, such as evaluating conflicting claims or producing explanations. 
These students could neither reach a conclusion nor were they able to determine the weaknesses of 
alternative solutions. In addition, these students were unable to change their actions or thinking; for example, 
they were not flexible in time management. These students somehow ‘over-analysed’ the problem, and, in 
the end, they failed in the problem-solving process. 
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In sum, the aspect that distinguished the participants were the differences in 1) aims, 2) the skills and 
disposition to think critically, 3) epistemological beliefs, 4) acquiring knowledge and 5) the skill of 
flexibility in adapting thinking and performance to the demands of the task. 


Critical thinking as a tool for enhancing understanding 
Generating personal understanding through ‘thorough processing’ 

Epistemological beliefs: Both objective and subjective knowledge can hold the 
highest epistemic status. Knowledge is fallible and contextual. 


Reaching a well-reasoned 
solution 

+ Figuring out how to complete 
multidimensional tasks and planning 
action 

+ Defining the problem, evaluating, 
go analysing, interpreting information, 
identifying alternative reasons, 

|S considering relationships between 
~ assumptions and ultimately reaching a 
.8 reasoned conclusion. 


.5 Reaching a limited 

SS solution 

3 

c5 - Identifying only a few ideas 
^ - Problems in evaluating, analysing and 

interpreting information 
- Problems in decision-making and 
reaching conclusions 
+ Searching for alternative ways to 
complete a task, changing one’s own 
routines or seeking help from 
authoritative specialists 


Endless weighing of the different 
options 

+ Defining the problem, identifying 
ideas, analysing, and interpreting 
information 

- Problems in time management, 

decision-making, reaching reasoned 
conclusions, evaluating knowledge ^ 
and judging the acceptability of 
information. gj 

- Problems in producing ^ 

explanations 3 


Problems in reaching a conclusion S’ 

rf- 

cr 

5 ’ 

- Identifying only a few ideas gj“. 

- Problems in evaluating, analysing era 
and interpreting information 

- Problems in decision-making, 
reaching conclusions and producing 
explanations 

- Expectation that a problem has a 
definite, right answer 

- Disposition to think critically may 
be low 


Critical thinking as a tool for determining truth or falsehood 
Seeking the right answer through ‘superficial processing’ 

Epistemological beliefs: Objective knowledge can reach the highest possible 
epistemic status. Knowledge is fallible. 


Figure 3. Summary of results. 


9. Conclusions 
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Even though the number of participants in this study was small, the variety in the students’ critical 
thinking was evident. Our results showed that after three years of university study, students’ critical-thinking 
skills and epistemological beliefs differed greatly, and eight out of ten volunteer students had some problems 
in critical thinking (cf. Arum & Roksa, 2011; Pascarella, Blaich, Martin & Hanson, 2011). The multi-method 
approach effectively revealed the variety of problems that university students may encounter. While many 
problems were related to the lack of disposition or skill, such as an inability to evaluate the credibility of 
documents, examine presuppositions, make interpretations, develop a personal perspective or generate 
arguments or conclusions, some of the problems were related to an inability to modify the whole critical- 
thinking process in a flexible manner. These findings corroborate the ideas of Fisher (2011) and Scheffler 
(1965), who suggested that individuals may be more or less skilled at different critical thinking abilities. In 
other words, a student may have the ability to identify and evaluate information, for example, yet at the same 
time struggle with other abilities, such as arriving at a conclusion, adjudicating conflicting claims or 
producing arguments. Therefore, it is clear that unilateral instructions concerning critical thinking are 
difficult to provide. 

The findings of this study support the idea that students’ epistemological beliefs were interwoven 
into their critical thinking (cf. Kuhn, 2005; Kuhn & Weinstock, 2002). Critical thinking emerged as a tool for 
understanding and determining the relevance and reliability of knowledge. Students who showed superficial 
processing believed that objective knowledge (i.e. scientific and verified knowledge) has the highest possible 
epistemic status. Although it is sensible to trust in scientific and empirical knowledge more than in personal 
opinions, the problem was that these students accepted scientific knowledge without question: they did not 
analyse, evaluate or interpret the information contained in the documents they were given. They acquired 
knowledge by appealing in equal measure to authoritative opinion, trusting in verified empirical evidence 
and listening to testimonies. These students palmed off a justification for knowing on authoritative experts. 

Contrary to the results of many previous studies (e.g. Kaartinen-Koutaniemi & Lindblom-Ylanne, 
2012; King & Kitchener, 2004; Kuhn, 1999, 2005; Kuhn & Weinstock, 2002), our main finding was that the 
students who appealed to authorities, testimonies or empirical evidence did not believe that knowledge is 
absolutely certain or unquestionable. Nor did these students share the view that beliefs accurately represent 
or correspond to reality. In effect, the students did not share a sense of metaphysical realism. Instead, these 
students claimed that scientific theories are uncertain, but probably true. The findings also show that the 
students who believed that knowledge is contextual and relative did not share a relativist view of knowledge. 
This finding is also contrary to the findings of earlier studies (e.g. Lahtinen & Pehkonen, 2012). Conversely, 
all of the students saw knowledge as fallible. The students believed that it is possible to seek criteria for 
evaluating, comparing and justifying beliefs or theories. Although some students struggled with evaluating 
knowledge, all of them saw current conceptions and theories as a starting point for further inquiry. They 
were thus fallibilist in the epistemological sense. 

This study further shows that students’ belief in themselves as critical thinkers and knowers is not 
necessarily equivalent to how they perform. Thus, we assume, along with previous studies (Elby & Hammer, 
2001; Greene & Yu, 2014), that the self-reported assessment method is not enough to gauge these kinds of 
complex processes. The present small-scale qualitative study has provided a unique picture of the critical 
thinking and personal epistemological beliefs of ten third-year bioscience students. Furthermore, this study 
has educational significance by revealing problems in these students’ critical-thinking skills and by 
describing the role of students’ conception of knowledge in the process of thinking critically. Through a 
multifaceted approach, it was also possible to deepen understanding of the emphases and gaps in the 
prevailing empirical research on critical thinking and personal epistemology. However, the findings of this 
study should not be interpreted as an accurate prediction of the target population. The findings of this study 
rather illustrate the nature of the phenomenon being studied, and how the different aspects of critical 
thinking and epistemological beliefs are intertwined and contribute to it together. This study involved a 
small, homogeneous sample of students in one discipline only. Owing to these limitations, more 
communication between the theoretical, empirical and methodological perspectives is required to increase 
understanding of this complex phenomenon in the different spheres. 
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Keypoints 


In this exploratory study we provide a multidimensional framework for analysing critical 
thinking by combining theoretical aspects from philosophical, educational and psychological 
approaches. 

3 In this exploratory study we provide a multidimensional framework for analysing critical 
thinking by combining theoretical aspects from philosophical, educational and psychological 
approaches. 

&■ A large body of data for each participant (n=10) was collected using multiple methods, 
including think-aloud protocol, interviews and a Collegiate Learning Assessment (CLA) 
performance task. 

3 The result shows that students’ epistemological beliefs were interwoven into their critical 
thinking: students used critical thinking as a tool for enhancing understanding and for seeking a 
right answer 

fr None of students shared an absolutist view of knowledge. 

None of students shared a relativist view of knowledge. 

All students shared a fallibilist view of knowledge. 
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Footnotes 

1 King and Kitchener (2004) do not call the lowest level of reflective thinking realism. However, in their 
model they maintain that, at the most limited level of thinking, knowledge is certain and is obtained from 
direct observation (p.7). This position fits metaphysical realism. 
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